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Abstract 


This study assesses the psychometric properties of a newly tested self-report assessment tool for media literacy, based on the twelve 
new media literacy skills (NMLs) developed by Jenkins et al. (2006). The sample (N—327) consisted of normal volunteers who com¬ 
pleted a comprehensive online survey that measured their NML skills, media exposure, digital participation, and civic engagement. A 
factor analysis performed on the survey items yielded 10 NMLs that emerged as significant subscales. A series of multivariate analyses 
of variance indicate a strong relationship between these NMLs and respondents’ exposure to new media forms, their participation in 
Web 2.0 platforms, and their civic engagement. Specifically, individuals who consume and produce new media extensively had the 
highest NML levels; Facebook, Twitter, YouTube and blogging emerged as the most significant factors in this analysis. The reliability 
and scalability of this assessment tool are discussed, in the context of current challenges facing media literacy evaluation. 
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To successfully navigate today’s data-rich and 
complex media environment, the ability to effectively 
find and evaluate necessary information is key. In 
order to function as critical consumers, it is imperative 
to be aware of the various types of bias inherent in 
specific media sources and communication channels 
(Covington 2004). Nonetheless, one must be more than 
an adept media consumer. The proliferation of Web 
2.0 platforms and new communication technologies 
is gradually turning consumers into producers, and an 
active and proficient involvement in the original crafting 
and dissemination of media messages is becoming a 
defining characteristic for today’s successful media 
users. Considering the novel skills that are required 
for a full participation in today’s communication 
environment, the concept of media literacy becomes 
an increasingly valuable asset, and an important 
prerequisite to both critical media consumption and 
responsible media production. A highly multifaceted 
and often debated term, media literacy is generally 
defined as “the ability to access, understand and create 
communications in a variety of contexts” (Buckingham 
2005), and refers to both traditional media channels 
(television, radio, print media) and new media, such as 
the Internet, mobile telephony and video games. 

Buckingham (2005) is right in noting that, 
just like print literacy, media literacy “should not 


be seen as a purely cognitive, rational affair: it also 
involves emotional response, enjoyment and cultural 
appreciation.” However, this type of engagement is 
often overlooked in media literacy theory. Furthermore, 
the traditional understanding of the concept does not 
always account for the active production of original 
media. In fact, in terms of net impact, creative media 
production has been shown to account for higher levels 
of media literacy than consumption-based practices 
(Phang and Schaefer 2009). This is not a surprising 
finding, given that, from a cognitive perspective, an 
increased participatory involvement facilitates deeper 
learning and stimulates critical thinking (Blumenfeld, 
Kempler and Krajcik 2006). 

But what exactly are these necessary media 
literacy skills? In their influential White Paper for the 
MacArthur Foundation, “Confronting the Challenges 
of Participatory Culture: Media Education in the 21st 
Century,” Jenkins et al (2006) make a critical theoretical 
contribution by identifying twelve “new media literacies” 
(NMLs) that are necessary for a full participation in 
today’s media environment. These skills are: play, 
performance, simulation, appropriation, multitasking, 
distributed cognition, collective intelligence, judgment, 
transmedia navigation, networking, negotiation, 
and visualization (Jenkins et al. 2006). The NML 
framework differs from previous media literacy models 
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in that it envisions people as active participants in the 
new digital environment; their role is reconfigured to 
account not only for media consumption, but for active 
media creation as well. Furthermore, in Jenkins’ view, 
these NMLs are “social and cultural competencies” that 
go beyond access to technology and proficiency with 
different media platforms; rather, they are conceived as 
critical skill sets that are bred and enhanced by one’s 
digital involvement in a participatory culture. 

A review of the literature in the field of media 
literacy points to the lack of an appropriate measurement 
tool to assess these new media literacy skills, especially 
among youth (Buckingham 2005). Certainly, existing 
tools do not capture the full spectrum of skills and 
propensities suggested by Jenkins et al. (2006). With 
increasing urgency, researchers and educators alike 
have pointed to the need to create national standards of 
media literacy assessment (Scharrer 2003; Potter 2004), 
which would provide empirical bases for evaluating 
the relationships between media literacy and digital 
participation, and for facilitating future media literacy 
interventions. 

Within the context of methodology development, 
the notion of media literacy is oftentimes employed in an 
arguably limited fashion, to refer principally to people’s 
critical understanding or processing of media messages, 
while overlooking their creation of media. Thus, the 
few studies that attempt a practical assessment of media 
literacy skills have consisted of measuring exclusively 
the ability to understand written and audiovisual texts 
(Dorr, Graves and Phelps 1980; Quin and McMahon 
1995; Phang and Schaefer 2009). Beyond the limited 
conceptual scope of such undertakings, these studies 
also point to the problematic nature of quantifying 
critical comprehension, given that such assessments 
are “implicitly normative” and favor specific readings 
of media messages as correct and “socially validated” 
(Buckingham 2005). Rosenbaum (2007) attempts the 
most ambitious and comprehensive effort to establish 
a standardized media literacy quantitative assessment 
tool, yet her instrument does not include any questions 
related to respondents’ production or dissemination 
of digital media, which is a major conceptual gap. 
Furthermore, her suggested assessment questions 
place a strong emphasis on the comprehension and 
interpretation of news programming, at the expense of 
other social and cultural practices of media consumption 
and production. 

Previous attempts to assess media literacy have 


mostly been centered on self-reported measures, due to 
the difficulty of experimentally gauging media literacy 
levels across different platforms and various media 
content. Furthermore, the reliance on qualitative data 
- which is typical of most studies in this field - means 
that such assessment projects are not feasibly replicable 
with larger groups (Rosenbaum 2007). In addition, in 
terms of empirical scope, most evaluative research in 
this field is concerned with assessing the effectiveness 
of various media literacy programs, especially in 
educational settings (Gonzales, Glik and Davoudi 
2004; Flobbs and Frost 2003; Phang and Schaefer 
2009), and there has been little effort to quantify 
media literacy at baseline levels prior to media literacy 
program onset (Rosenbaum 2007). Additionally, due to 
the predominantly educational applications of media 
literacy programs, assessment projects have failed to 
measure media literacy levels across non-youth, in 
addition to youth, populations (Maness 2004; Brown 
1991). 

The present study aims to address these 
methodological lacunae by developing and validating 
a comprehensive assessment tool that could be 
used to measure new media literacies (NMLs) in 
both adult and juvenile populations. Built around 
the twelve NML skills identified by Jenkins et al. 
(2006), this questionnaire assumes a multi-component 
understanding of media literacy, tackling both the 
consumption of media messages, as well as the original 
creation of multimedia material. The items on the 
questionnaire have carefully been crafted to address 
both online and offline behaviors, in accordance to the 
NML framework, which views new media literacies as 
social and cultural skill sets. 

In assessing the psychometric properties of this 
new assessment tool, survey data will first be factor 
analyzed and an assessment of the reliability of the 
measure will be made. We will compare these factors 
with Jenkins’ original 12 NML skills and discuss the 
overlaps and differences. Then, to provide an initial 
assessment of validity, the scale(s) will be correlated 
with several variables that media literacy should 
predict, such as one’s degree of digital participation, the 
amount of time spent with different forms of media, or 
the extent to which users engage in creative multimedia 
projects. Finally, we will also perform an exploratory 
analysis of the impact of various demographic factors 
on the components of media literacy, as they emerge 
from the factor analysis. 
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If the survey instrument is accurately 
constructed, we expect to see 12 separate subscales 
- similar to the 12 NMLs identified by Jenkins et al. 
(2006) - resulting from the factor analysis. In terms of 
the relationship between media exposure and NMLs, 
we hypothesize that higher levels of new media 
literacies will predict a higher degree of engagement 
with media forms - particularly new digital media - 
and that there will therefore be a significant difference 
in NMLs between people with low versus high levels 
of media exposure. An increased degree of digital 
participation in various Web 2.0 platforms should also 
relate to high NML levels, with light users scoring 
lower in media literacy than heavy users of these digital 
platforms. We also hypothesize that high NML levels 
should predict a greater propensity for multimedia 
creation, and, respectively, civic engagement. Finally, 
in terms of demographic differences, we expect young, 
educated and relatively wealthier demographic groups 
to show higher NML skills than users that are older, less 
educated, or from lower socio-economic strata. 

Methods 

Sample 

The sample for this study (N=327) was a 
convenience sample of normal volunteers over the 
age of 18. In terms of gender distribution, the sample 
contained 131 male respondents and 187 female 
respondents. The average age was 33.7 years (SD=11.7). 
In regards to ethnicity, 83.9% of respondents were 
white, and 77.3% indicated English as their primary 
language. Income and education levels were normally 
distributed. 

Recruitment 

The questionnaire was made available on the 
Qualtrics website, and the link was distributed via 
email and social networking sites, primarily Twitter 
and Facebook. There were no material incentives 
for taking part in this study; however, to maximize 
participation, the questionnaire was designed as a fun 
personality quiz, and users received their personalized 
media literacy score at the end of the survey, as well as 
a description of the type of media user they were, based 
on their responses to the questionnaire. 

Survey Design 

The survey (see Appendix A) was structured 


around 4 main sections: demographics, media use habits, 
new media literacies (NMLs), and civic engagement. 
None of the questions on the survey were mandatory, 
and respondents could choose to skip any question. To 
maximize the validity of the findings, the order of the 
questions within each section was randomized, so that 
each participant received them in a different order. 

The third and most crucial section aimed 
to assess respondents’ new media literacy skills 
(NMLs) by presenting them with a randomized series 
of 60 statements about their personality, social and 
cultural modes of engagement, online and offline peer 
interaction, learning styles, and media consumption and 
creation patterns. The statements were conceptually 
built around the 12 NML skills identified by Jenkins et 
al. (2006). These are: 

• Play — the capacity to experiment with one’s 
surroundings as a form of problem-solving 

• Performance — the ability to adopt alternative 
identities for the purpose of improvisation and 
discovery 

• Simulation — the ability to interpret and construct 
dynamic models of real-world processes 

• Appropriation — the ability to meaningfully sample 
and remix media content 

• Multitasking — the ability to scan one’s environment 
and shift focus as needed to salient details 

• Distributed Cognition — the ability to interact 
meaningfully with tools that expand mental 
capacities 

• Collective Intelligence — the ability to pool 
knowledge and compare notes with others toward 
a common goal 

• Judgment — the ability to evaluate the reliability 
and credibility of different information sources 

• Transmedia Navigation — the ability to follow 
the flow of stories and information across multiple 
modalities 

• Networking — the ability to search for, synthesize, 
and disseminate information 

• Negotiation — the ability to travel across diverse 
communities, discerning and respecting multiple 
perspectives, and grasping and following alternative 
norms 

• Visualization - the ability to create and understand 
visual representations of information. 

To ensure an adequate factor analysis while 
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minimizing the duration of the survey, we decided to 
include 5 statements for each NML, for a total of 60 
questions. These statements addressed both technology- 
related and non-technology-related behaviors, in 
accordance with our view that the NML skills are 
social and cultural competencies that stretch beyond 
media expertise or technological capability. The 
questions were assessed on a 5-point Likert-type scale 
(l=Strongly Disagree, 5=Strongly Agree). For a full list 
of the statements used in the questionnaire, please see 
Appendix A. 

Results 

New Media Literacies (NMLs) Scale: Factor Analysis 
and Subscale Reliability 

Although all of our scale items collectively 
attempt to measure new media literacy levels, and 
the overall reliability of the scale is high (Cronbach’s 
a=.903), we were interested in identifying the specific 
subcomponents that make up this concept. Our initial 
research question was whether the subscales of this 
survey instrument map well onto Jenkins’ 12 NMLs. 
Specifically, we were interested in seeing if, as predicted, 
the scale would break down into components that were 
similar to those identified by Jenkins et al. (2006). To 
address this question, we performed a factor analysis on 
the 60 items, and then assessed the reliability of each 
separate subscale that emerged from the factor analysis. 

The Kaiser-Meyer-Olkin test of sampling 
adequacy was .824, much above the recommended value 
of .6, and Bartlett’s test of sphericity was significant 
at 5,325 (p < .001), which means that the items are 
appropriate for factor analysis. We therefore performed 
a principal component factor analysis, using the 
Varimax method of rotation with Kaiser normalization. 
To simplify the results of the analysis, we decided to 
suppress items whose coefficient was below 0.5. The 
factor analysis performed in SPSS yielded 17 factors 
with an eigenvalue above 1.00; these accounted for 61% 
of the variance. In descending order by variance, the 
subscales that emerged were: negotiation, networking, 
judgment, play, multitasking, appropriation, transmedia 
navigation, visualization, distributed cognition, and 
performance. The reliability of all subscales was found 
to be satisfactory. 

In conclusion, out of the 12 NML skills that 
Jenkins identified as the competencies of media literacy, 


10 were identified in the factor analysis of our scale. 
This is a rather impressive and encouraging finding, 
especially given the fact that all 60 items of the scale 
were completely randomized and thus the items that 
made up each of these 12 subscales never appeared in 
order. The two NMLs that did not distinctly emerged 
from the factor analysis were collective intelligence 
and, respectively, simulation; rather than clustering 
together as distinct factor components, the items 
measuring these two dimensions ended up being spread 
out over the different subscales. 

Relationship between Media Use and NMLs 

Once the factor analysis revealed the various 
new media literacy skills (NMLs) that the scale 
constituted of, we proceeded to explore the relationship 
between these NMLs and patterns of media exposure 
and digital participation. NML subscale composites 
were formed by calculating the aggregate mean of their 
constituent items. We thus computed 10 new variables, 
which represented the specific factors that emerged as 
a result of the factor analysis: negotiation, networking, 
judgment, play, multitasking, appropriation, transmedia 
navigation, visualization, distributed cognition, and 
performance. 

Media Exposure 

To be able to see the variations in NML skills 
across different groups, depending on the intensity of 
users’ exposure to media, we ran multivariate analyses 
of variance (MANOVAs). For the purpose of facilitating 
the analysis, in this and subsequent MANOVA tests, 
respondents were dichotomized into two groups, using 
a median split. The condition of belonging to one of 
these two groups (low or high media use, light or heavy 
digital participation) was used as the dependent variable 
in the MANOVAs, while the 10 NMLs were considered 
as independent variables. 

We first looked at respondents’ cumulative 
media exposure, which included time spent with all 
forms of media: Internet, television, print media, and 
videogames. According to our second hypothesis, we 
expected to see a significant difference in NML skills 
between high and low media users. 

The multivariate difference in media literacy 
levels assessed using MANOVA was indeed significant: 
F(10, 316)= 3.025, p=.001, with avid media consumers 
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scoring higher across all NML skills (M=3.878, 
SD=.382) than less enthusiastic media consumers 
(M=3.694, SD=.397). The univariate differences 
between the high and low media exposure groups were 
particularly pronounced in the areas of negotiation (F(l, 
325)=7.130, p=.008), networking (F(l, 325)=12.058, 
p=.001), appropriation (F(l, 325)=7.220, p=.008), 
play (F(l, 325)=7.167, p=.008), multitasking (F(l, 
325)= 14.923, p<.001), and transmedia navigation (F(l, 
325)=10.403, p=.001). 

Next, we explored the relationship between 
NMLs and exposure to specific media. In terms of 
Internet use, there was a significant difference in media 
literacy, examining the multivariate effect between low 
and high users: F( 10,316)= 3.171, p=.001, with the most 
striking contrast occurring in terms of networking skills. 
Due to the interconnecting and socializing features of 
the Internet, less enthusiastic Internet users scored much 
lower in networking skills (M=3.859, SD=.654) than 
frequent users (M=4.201, SD=.546). For videogames, 
the difference between frequent and infrequent users 
was also significant (F(l, 316)=2.811, p=.002), with 
avid gamers scoring substantially higher (M=4.143, 
SD=.576) than their peers (M=3.875, SD=.601) in the 
domain of play, or experimental problem-solving (F(l, 
325)=17.019, p<.001) (Table 1). 


Dependent 

variable 

F-Value 

Most significant 
NMLs 

Media 

exposure 

(cumulative) 

F(10, 316) = 
3.025, p= .001 

negotiation, 
networking, 
appropriation, 
play, multitasking, 
transmedia navigation 

Internet 

F(10, 316) = 
3.171, p= .001 

networking 

Videogames 

F(l, 316) = 

2.811, p= .002 

play 

Table 1 . Signi 

leant effects for media exposure 


Our questionnaire addressed users’ exposure to 
four different forms of media: two new ones (Internet 
and videogames) and two old ones (television and 
print media). Interestingly enough, while the difference 
between light and heavy users of the Internet and 
videogames - i.e. new media - was substantial (F(10, 
316)= 3.171, p= .001 for Internet, and, respectively, F(l, 


316)=2.811, p=.002 for videogames), this difference 
was not significant in the case of traditional media: for 
television, F(10, 316)=1.516, p=.132, while for print 
media, F(10, 316)=1.285, p=.238. 

Digital Participation 

The second half of this section sought to 
quantify respondents’ level of digital participation and 
engagement with online platforms. According to our 
hypothesis, we expected to see a significant difference 
between high and low digital participation levels, with 
highly engaged users showing higher NML skills than 
less avid digital participants. 

Before proceeding to investigate the relationship 
between media literacy and specific Web 2.0 platforms 
(e.g. Facebook, Twitter, YouTube, blogging, etc), we 
first performed a general MANOVA, using respondents ’ 
cumulative digital participation reports - split along the 
median into light and heavy users - as our independent 
variable. The difference in NMLs between these two 
groups (users with high digital participation levels 
versus those with lower participation levels) was 
indeed significant; thus, across all platforms, F(10, 
316)=3.172, p=.001 (Table 2). 

A high degree of engagement with Facebook 
correlated strongly with high media literacy levels. 
With Facebook use - high or low - as our independent 
variable and the NMLs as our dependent variables, we 
performed a MANOVA and found that the difference 
between light Facebook users (including non-users) and 
heavy users was significant: F(10, 316)=5.294, p<.001. 
A subsequent univariate analysis revealed that the most 
pronounced differences in NMLs for infrequent versus 
frequent Facebook users appeared, not surprisingly, in 
the areas of networking (F(l, 325)=31.527, p<.001). 

Twitter also emerged as having a significant 
relation to NML skills. The multivariate difference 
between light (which, again, includes non-users) 
and heavy Twitter users was considerable: F(10, 
316)=3.181, p=.001. In terms of the specific NMLs, 
we found that light Twitter users scored lower in 
networking (F(l, 325)=26.908,p<.001) and transmedia 
navigation skills (F(l, 325)=9.128, p=.003) than more 
enthusiastic tweeters. 

The next consequential independent variable 
that we examined was YouTube; specifically, we 
were interested in whether light and heavy users of 
YouTube differed in NML levels, and if so, in what 
particular NML skills. Our MANOVA pointed to a 
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significant difference between light and heavy YouTube 
users: F(10, 316)=4.553, p<.001. The NMLs that 
YouTube fans excelled at were appropriation (F(l, 
325)=22.009, p<.001) and transmedia navigation (F(l, 
325)=20.352, p<.001), but also, to a less astounding 
degree, performance (F(l, 325)=15.845, p<.001) and 
negotiation (F(l, 325)=11.978, p=.001). 

Finally, blogging emerged as another particularly 
important platform in terms of NML skills. With 
blogging as our independent variable, and the NMLs 
as our dependent variables, we found a significant 
difference in NML skills between bloggers and non¬ 
bloggers: F(10, 316)=4.747, p<.001. Individuals who 
keep a blog scored much higher in appropriation (F(l, 
325)=27.681, p<.001) and networking skills (F(l, 
325)=27.409, p<.001). 


Dependent 

variable 

F-Value 

Most significant 
NMLs 

Digital 

participation 

(cumulative) 

F(10,316)=3.172, 

p=.001 

appropriation, 

networking, 

transmedia 

navigation 

Facebook 

F(10,316)=5.294, 

p<.001 

networking 

Twitter 

F( 10,316)=3.181, 
p=.001 

networking, 

transmedia 

navigation 

YouTube 

F(10,316)=4.553, 

p<.001 

appropriation, 

transmedia 

navigation, 

performance, 

negotiation 

blogging 

F(10,316)=4.747, 

p<.001 

appropriation, 

networking 

creating 

media 

projects 

F(10,315)=6.635, 

p<.001 

appropriation 


Table 2. Significant effects for digital participation 


While Facebook, Twitter, YouTube and 
blogging displayed the strongest connections to NML 
skills, we also found marginally significant differences 
between users and non-users of other social networking 
sites (such as Bebo, Friendster, or My Space), message 
boards, and online games. The data we collected for 
online group membership, multiplayer gaming and, 
respectively, podcasting, was not sufficient to allow us 


to explore the differences between groups of users and 
non-users. 

Finally, since creative ability is an inseparable 
part of media literacy - as discussed in the initial section 
of this paper - we also tested for the significance of the 
relationship between NMLs and multimedia creation. 
Indeed, as hypothesized, there was a strong difference 
between respondents who reported creating multimedia 
projects often and those who admitted engaging in such 
projects only rarely: F(10, 315)=6.635, p<.001. Not 
surprisingly, the top NML skill that made the difference 
here was appropriation (F(l, 324)=46.553, p<.001); 
frequent multimedia creators scored much higher in 
appropriation levels (M=3.915, SD=.758) than less 
enthusiastic producers (M=3.327, SD=.795). 

Civic Engagement 

In addition, because ah of the affiliations above 
are online or computer-based, we also included the 
element of civic engagement as an alternative, offline 
type of affiliation. As predicted in our hypothesis, 
respondents with high civic engagement levels had 
higher NML skills than those with low civic engagement 
levels; the difference was significant (F( 10,313)=3.516, 
p<.001), with networking, play and judgment emerging 
as important NMLs in this area (Table 3). 

Demographics 

Next, we examined whether individuals 
differing in gender, education and income varied in 
NML skills. Indeed, using MANOVA, we found that the 
NML skills of men and women differed to a significant 
extent: F(10, 307)=3.793, p<.001, with the statistically 
significant univariates occurring in the areas of play, 
transmedia navigation, and performance. Men proved 
to be better at play (F(l, 316)=8.250, p=.004) and 
performance (F(l, 316)=5.671, p= .018), while women 
excelled at transmedia navigation (F(l, 316)=7.167, 
p=.008). We also found that education and income had 
an effect on NML levels: F(10, 315)=2.411, p=.009 
for education and, respectively, F(10, 310)=2.031, 
p=.03 for income. In terms of the specific NMLs 
involved, we found that less educated people proved 
more adept at multitasking (F(l, 324)=6.191, p=.013), 
while more educated respondents showed higher levels 
of distributed cognition (F(l, 324)=6.505, p=.011). 
Interestingly, contrary to what we had expected, age 
did not make a significant difference in NML skills: 
F(10,301)=1.644,p=.094. 
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Discussion 

Summary of Findings 

The study was designed to test the validity of 
a newly developed survey instrument measuring new 
media literacies (NMLs), in accordance to the theoretical 
framework proposed by Jenkins et al. (2006), as well 
as to explore the connections between individuals’ 
NML levels and their degree of media exposure, digital 
participation, and civic engagement. Our first hypothesis 
was that the survey instrument would be able to be 
broken down into separate components similar to the 
12 NMLs identified by Jenkins. Indeed, in the current 
work, we found that the media literacy measure, when 
factor analyzed, almost fully mapped onto Jenkins’ 
classification of NMLs: 10 of the NMLs emerged as 
distinct components in the factor analysis, with only 
2 NMLs failing to appear as significant subscales in 
the survey. Furthermore, we found that each of these 
subscales had adequate reliability. 

In line with the second hypothesis, we found 
that higher levels of new media literacy skills indeed 
predicted an increased degree of exposure to media; 
however, this was only true of new media (Internet and 
videogames), and not traditional media (television and 
print media) as well. This is an interesting conclusion, 
which supports the view that new digital media, due 
to their interactive and highly socializing nature, 
are more adept at breeding the social and cultural 
competencies needed for a full participation in today’s 
digital environment than traditional media, which are 
inherently more passive (Jenkins et al., 2006). 

In terms of digital participation, we hypothesized 
that higher levels of media literacy should predict a 
higher degree of engagement with Web 2.0 platforms, 
as well as an increased propensity for multimedia 
creation. This hypothesis was fully supported. Out of 
the digital platforms we explored in this study, the ones 
that emerged as particularly significant in this analysis 
were Facebook, Twitter, YouTube, and blogging. For 
Facebook, the difference between light and heavy users 
was especially pronounced in the area of networking, 
with enthusiastic Facebook users displaying extremely 
high networking skills. This result is unsurprising, 
given the function of Facebook as a social networking 
site, but this connection is important in regards to the 
applicability of such online-learned skills in the context 
of one’s offline behavior. For Twitter, the two main NMLs 
where light and heavy users significantly differed were 


networking and transmedia navigation; this conclusion 
makes sense, and can be explained by the hyperlinked 
and social nature of the Twitter platform. YouTube 
also emerged as an extremely significant platform 
in terms of NML skills, with the main differences 
between light and heavy users occurring in the areas 
of appropriation, transmedia navigation, performance, 
and negotiation. These results are most likely explained 
by the primary functions of the YouTube platform as a 
crucial depository of popular culture clips (to be used 
in appropriation processes), a source of multimedia 
information (encouraging transmedia navigation), a 
democratic limelight for stardom and personal opinion 
(performance) and a transnational hub that facilitates 
intercultural learning (negotiation). For blogging, the 
NMLs that made the most significant difference between 
bloggers and non-bloggers were appropriation and 
networking. Most likely, this is due to the increasingly 
interlinked nature of the “blogosphere”, with writers 
linking to other blogs of interest, keeping a blogroll on 
their personal page, republishing relevant posts, etc. 
This process of hyperlinked interconnectedness, while 
gradually transforming the personalized “blogosphere” 
into one global community, increasingly requires 
networking and appropriation skills that allow one to 
most effectively tap into this informal community. 

The results of this study also supported the 
connection between multimedia creation and media 
literacy; as hypothesized, higher NML levels predicted 
a propensity for multimedia creation, and the difference 
between frequent and infrequent digital creators was 
extremely significant. This is in line with the literature 
in the field, which claims that the ability to creatively 
produce and distribute multimedia texts should 
correlate strongly with higher levels of media literacy 
(Phang and Schaefer 2009). 

Similarly, the results also confirm the 
connection between new media literacies and civic 
engagement, which is emerging as a critical application 
of NML educational initiatives (Rheingold 2008). Our 
hypothesis regarding the positive relation between 
media literacy and civic engagement was fully 
supported, with respondents that scored highly across 
the NMLs showing much higher degrees of civic 
engagement than their less media literate peers. 

Finally, in terms of demographic differences, 
our hypothesis was partially supported. While education 
and socioeconomic status did make a difference in 
terms of NML skills, the level of significance here was 


22 


I. Literat / Journal of Media Literacy Education 6:1 (2014) 15-27 


not as high as we had expected. Age, on the other hand, 
which we had predicted to be a crucial independent 
variable in our analysis, did not emerge as a significant 
factor. While this finding is certainly at odds with the 
general perspective of the field (Buckingham 2005) - 
and especially since the present survey included a large 
variety of youth culture questions - perhaps the lack of 
significance can be explained by the fact that the older 
people that took this survey were recruited principally 
through professional (academic) relationships, which 
meant that they are highly connected to the field of 
media and communication because of their education, 
interests, or professional experience. 

In conclusion, as evidenced by the support for 
our main conceptual hypothesis, the data gathered in 
this study will be instrumental in perfecting a validated 
quantitative assessment tool to complement NML 
initiatives built around this particular framework. 
So far, educational endeavors aimed at cultivating 
these skills were generally assessed using qualitative 
evaluation tools, which are inherently unfit for use with 
large samples, and are much harder to implement due 
to logistical considerations. We therefore hope that this 
questionnaire, especially used as a baseline measure of 
new media literacies, will help provide a more accurate 
and comprehensive picture of individuals’ abilities in 
this domain. 

Furthermore, the study provided critical 
information about the connections between new media 
literacies (NMLs), media exposure, and engagement 
with different Web 2.0 platforms; this represented 
a much-needed addition to the literature on media 
education, which so far did not address these specific 
correlations. In terms of the validity of the present 
assessment tool, the fact that our hypotheses regarding 
the connection between media literacy and media 
use habits were strongly supported lends additional 
predictive validity to this survey instrument. This is a 
highly significant conclusion that adds further import 
to the current study. While the causal relationships 
between these variables would need to be examined 
longitudinally, over time, it is our interpretation that the 
relationship between media use and media literacy is 
a circular one, involving a virtuous feedback loop: for 
instance, while extensive use of the Internet raises one’s 
media literacy levels, media literate individuals are also 
more likely to access the internet considerably more. 


Limitations 

A major shortcoming of the present study 
concerns the recruitment of the sample used. 
Specifically, subjects were recruited through email, 
Facebook, Twitter and other social networking sites, 
which implies an inherent bias towards individuals 
that already have a certain degree of media literacy. 
Furthermore, due to the personal connections of 
the author, some of the respondents are part of the 
academic community - many of them studying media 
or communication - which also means that they are 
considerably more media literate than the general 
population. 

The content of the survey can also be improved 
for future use. Since the major aim of this study was 
to test the validity and usability of this questionnaire, 
we provided a comments section at the end of the 
survey, where respondents were able to provide their 
feedback on this particular assessment tool. Following 
an analysis of these comments, we found that a few 
participants thought that the survey was excessively 
youth-focused. Given that the survey will be used 
primarily for assessment in educational contexts, and 
was thus designed with this application in mind, and 
since the average age of the respondents in this study 
was 33.7 years, this is a very valid comment, which 
points to the need to develop distinct versions of this 
assessment tool, depending on the characteristics of 
the target population. An improved version of this 
questionnaire should also attempt to condense the 
length of the survey in order to reduce the dropout 
rate. The present survey contained 80 questions and 
took about 20 minutes to complete; while this time 
commitment is still much more manageable than in the 
case of qualitative assessments (which generally take 
much longer), it should be shortened for future use. We 
will need to review the questions measuring simulation 
and collective intelligence - the two NMLs that did not 
emerge in the factor analysis - in order to make sure 
that they are properly evaluated in this assessment tool. 

Implications for Future Research 

Considering the inherent shortcomings of using 
quantitative methods to measure subjective skills and 
competencies, this study invites important questions 
about the applicability of quantitative methodologies in 
skills testing .Future research in the area of measurement 
and evaluation should explore the need to triangulate 
such quantitative measures with qualitative evaluations, 
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in order to achieve a fuller and more precise assessment valuable starting point in this direction, and a much- 

of media literacy. For instance, a survey such as this needed inquiry into the challenges facing media literacy 

one can be combined with practical workshops, where assessments in both national and international contexts. 

participants are able to actually demonstrate the skills 

that they report having. There is also a need to better 

understand the relationship between baseline and 

endline evaluations. Especially in the case of program 

evaluations, it is crucial to determine whether surveys 

such as this one can be used for both baseline and 

endline assessments, and specifically, what particular 

changes need to be made in order to best underscore the 

differences in media literacy levels between these two 

points of evaluation. 

A consequential direction for future research 
in this field concerns the relationship between specific 
digital platforms and new media literacies. Some of 
the comments we received in the feedback section of 
the survey confirm this need for further clarification. 

“Your test assumes that particular media makes you 
more media literate”, said one user, while another 
commented that “you’re assuming that not engaging in 
certain media activities is due to lack of literacy rather 
than a personal choice. Maybe it just doesn’t interest me 
or it runs counter to some other value or interest I might 
have.” These are extremely valuable and warranted 
critiques, given that our conceptual framework of the 
NMLs considers these platforms as central to one’s 
media literacy skills in today’s digital environment, and 
thus assumes a strong connection between engagement 
with these tools and media literacy levels. 

Finally, future work in the field of media 
literacy assessment should attempt to tackle the issues 
of standardization and scalability. Can a survey such as 
this one be used universally, with different populations 
and in different types of program evaluations? While 
a national (or perhaps even international) standard of 
media literacy evaluation would certainly be useful and 
instrumental in educational endeavors, our experience 
with this study indicates that an over-standardization 
of such assessment tools can lead to a decrease in the 
validity of the findings. Particularly, age and cultural 
background emerge as two critical factors in the tailoring 
of such surveys to best assess particular populations. We 
therefore hope that our study has helped to underscore 
the importance of achieving a proper balance between 
universal applicability and personal relevance. While 
further research is certainly needed regarding the 
feasibility of quantitative methods of assessment in 
the field of media literacy, we believe our study is a 
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APPENDIX A: The NML Questionnaire 

Part 1: Demographic Information 

Sex: M/F 
Age: 

Education: Some high school/ Completed high school/ Some college/ Completed college/ Masters Degree/ PhD/ Other 
higher education degree 

Income: Less than 10,000/ 10,000-20,000/ 20,000-30,000/ 30,000-40,000/ 40,000-50,000/ 50,000-60,000/ 60,000-70,000/ 
70,000-80,000/ 80,000 or more 

Ethnicity: White/ Hispanic/ African-American/ Asian/ Native American/ Pacific Islander/ Other 
Primary language spoken at home: English/ Spanish/ Korean/ Chinese/ Russian/ Other (specify) 

Part 2: Digital Participation 

Do you have a computer at home? Y/N 

Do you have Internet at home? Y/N 

How many hours a week do you generally spend: 

a. On the Internet: 

i. for school or work 

ii. in your free time 

b. Watching TV (not on your computer) 

c. Reading books, magazines or print newspapers 

d. Playing games (online, on your cell phone, on PlayStation, Wii, Xbox etc.) 

On average, how many hours a week do you spend on... 

a. Facebook 

b. Twitter 

c. YouTube 

d. MySpace/Bebo/Friendster/other social networking sites 

e. Online groups (Yahoo Groups, Google Groups, other online communities) 

f. Message Boards 

g. Games (online, on your cell phone, on PlayStation, Wii, Xbox, etc): 

a. By myself 

b. With other players 

h. Blogging (Blogspot, Wordpress, Blogger, etc.) 

i. Podcasting 

j. Other online activities (specify) 

How often do you create projects that use video, audio, music, photographs, etc. outside of school or work, in your free 
time? 

a. Often 

b. Sometimes 

c. Rarely 

d. Never 

Are you familiar with the concept of “media literacy”? 

a. no 

b. yes 

If so, in your own words, how would you define “media literacy”? 

Part 3: Assessing the 12 NML skills 

NOTE: For all the questions below, the possible answers were: Strongly Disagree, Disagree, Neither Agree or Disagree, 
Agree, Strongly Agree. 

The order of all questions in this section was randomized. 

PLAY: 

I have taught myself something new on a computer by seeing what happens when I play around with it. 

When I have a new cell phone or electronic device, I like to try out all the buttons to see what they do. 
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I enjoy taking things apart and then putting them back together to find out how they work. 

When I am faced with a problem, I usually try out a few different ways of solving it before I give up. 

When I get stuck trying to solve a problem, I see it as a learning opportunity rather than a personal failure. 

SIMULATION: 

I try to put myself in other people’s shoes to understand their problems or situations. 

It is important to have simulations of dangerous events like earthquakes or safety evacuations, so that people know what to 
do in a crisis. 

I appreciate simulation games and activities like Second Life, SimCity, The Sims, FIFA, Tiger Woods PGA Tour, etc. 

I think about the way in which reality is represented in movies with computer-generated simulation, like Avatar, Inception, 
300, Sin City, Iron Man, X-Men, etc. 

1 would like to participate in a simulation of something I cannot experience in real life, like flying a space shuttle to the 
moon, or piloting a fighter jet. 

PERFORMANCE: 

1 have often taken on a different identity in order to experience something new or to solve a problem (online games, role- 
playing, theatre exercises). 

I know what an avatar is. 

I feel I am a different person online than how I act in person. 

In certain situations, it is necessary to not be yourself. 

Actors learn a lot about life from the roles they play in films and on stage. 

APPROPRIATION: 

I have incorporated other people’s public work to create my own piece of art, like mixing music tracks, making an art 
collage, or stringing together video clips. 

I have created something new that incorporates stuff from popular culture, like writing a short story based on a character in 
my favorite book, making a fan video, or a music remix. 

When doing a creative multimedia project, I don’t think it is wrong to take samples from my favorite artists’ songs or videos. 
If I would make a fan video about my favorite celebrity or artist or band, they’d probably be happy if they found out about it. 
It is important for young people to learn how to use stuff from popular culture in their own creative ways. 

DISTRIBUTED COGNITION 
I don’t agree that smart people are bom smart. 

My environment plays a big part in how smart I am. 

I have to keep learning from my surroundings in order to become smarter. 

I'm usually pretty good at knowing what do to do or who to ask if I want to find out more about a specific topic. 

I find it important to use tools like spell check, a calculator, encyclopedia, etc to help me in my learning or work. 

MULTITASKING 

1 manage to do my work successfully while doing other things like listening to music or texting. 

I can usually prevent getting distracted and focus on 

tasks well when other things are happening around me, like people talking, TV, music, internet, etc. 

When I work on my computer, I like to have different applications open in the same time. 

My generation is good at multitasking, i.e. doing several things at once. 

I don’t think anybody should give me a hard time if I feel I can work on several things at once. 

COLLECTIVE INTELLIGENCE 
1 enjoy working with others on projects or assignments. 

When I can’t solve a problem or find a piece of information by myself, I use the internet or social media to connect with 
others and find what I am looking for. 

I enjoy the collaborative aspect of things like Wikipedia, team games, online fan communities, community message boards, 
etc. 

I think I can learn a lot from my friends. 
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I don’t think it’s a sign of weakness or stupidity to ask a friend or a colleague for help on work assignments or other problems. 


JUDGMENT 

I can effectively determine whether or not the information I find online is correct and reliable. 

When I’m interested in a topic, I gather information from a bunch of different sources (like TV, radio, the internet, etc) to 
try to get the full picture. 

When 1 search for something online and 1 get thousands of results, I can effectively decide which ones will be the most 
useful for me. 

I am able to enter the right words in a search engine to find what I am looking for. 

I can identify prejudice or bias in media (e.g. racism on certain websites, prejudice against women in song lyrics, etc). 
TRANSMEDIA NAVIGATION 

1 follow my favorite shows, actors, musicians etc across different platforms and media (TV, magazines, internet, Facebook, 
Twitter, etc). 

1 can imagine the same story being told in different ways, such as through music, acting, writing, drawing, etc. 

I often visit the websites (either official or fan-created) of my favorite TV shows, bands, etc. 

If I am curious about something I saw on TV, I will check it out online later. 

I’m happy that I can learn about my favorite things in different ways (on TV, on the internet, on Facebook, etc) 
NETWORKING 

I think that reading other people’s recommendations on sites like Amazon or Yelp is useful in helping me make decisions. 

I like to share my favorite links or creative work on social media sites like Facebook or YouTube or Twitter. 

I often share links on Facebook, Twitter, my blog, etc. 

When I go online, I like to feel like I am part of a community. 

It is important for me to be able stay in touch with my friends online too, and not only in real life. 

NEGOTIATION 

My experience on the internet and/or in videogames has made me more understanding of those different from myself. 

1 think the internet offers a very important opportunity to get to know people from different backgrounds and different 
places. 

I am happy that I can interact online or on Facebook with people from all over the world. 

I have learned something new about another culture from surfing the internet, playing online games, participating in online 
communities or forums, etc. 

I think that using the internet and/or playing videogames makes people more open to other cultures. 

VISUAFIZATION 

1 feel I understand things better when I can think of them visually. 

When I prepare a project for work or school, I like to use as many images, graphs and diagrams as possible. 

I think I am pretty good at understanding information from images, graphs, diagrams and other visual tools. 

I like the fact that I can see all my friends on my Facebook profile. 

I find Google Maps and/or Google Earth to be extremely useful tools. 

Part 4: Civic Engagement 

NOTE: For all the questions below, the possible answers were: Strongly Disagree, Disagree, Neither Agree nor Disagree, 
Agree, Strongly Agree. 

I believe I can make a difference in my community. 

Being actively involved in national, state and local issues is my responsibility. 

1 have volunteered in my community. 

I have done something to help raise money for a charitable cause. 

I stay informed on current events and politics. 



